feat(observability): add SWIFT networking user journey monitoring (ARO-25382) - #6359
feat(observability): add SWIFT networking user journey monitoring (ARO-25382)#6359Andrew Denton (ventifus) wants to merge 2 commits into
Conversation
|
Hi Andrew Denton (@ventifus). Thanks for your PR. I'm waiting for a Azure member to verify that this patch is reasonable to test. If it is, they should reply with Tip We noticed you've done this a few times! Consider joining the org to skip this step and gain Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
6049995 to
98bedfc
Compare
|
|
||
| ```bash | ||
| # Restart mgmt-agent to force a resync of NIC capacity | ||
| kubectl rollout restart deployment -n mgmt-agent |
There was a problem hiding this comment.
This command is missing the deployment name .kubectl rollout restart deployment -n mgmt-agent will restart all deployments in the mgmt-agent namespace, not just the intended one. Confirmed via mgmt-agent/deploy/templates/deployment.yaml that the actual deployment name is mgmt-agent. Should be:
kubectl rollout restart deployment mgmt-agent -n mgmt-agent
|
|
||
| # Mitigation | ||
|
|
||
| ## Scenario: Stuck MTPNC deletion (MTPNC has DeletionTimestamp, finalizer not clearing) |
There was a problem hiding this comment.
Good context and fix options here, with a nice Risk callout. Per the TSG template, this scenario should also explicitly state Goal and Preconditions before the fix options, so the SRE knows why they’re doing this and when it’s safe to apply.
Suggested:
Goal: Unblock a MultiTenantPodNetworkConfig stuck in DeletionTimestamp with a finalizer DNC-RC never cleared, so the router pod can be recreated.
Preconditions: Confirmed the MTPNC has a non-null DeletionTimestamp and the finalizer is still present (per the chain described above); DNC-RC logs show no active reconciliation attempt in the last 10+ minutes.
|
|
||
| > **Risk:** Draining a node evicts all pods on it. Confirm no critical workloads are pinned and check PodDisruptionBudget constraints before proceeding. | ||
|
|
||
| ## Scenario: mgmt-agent reporting zero SWIFT NICs on a node |
There was a problem hiding this comment.
Missing Goal and Preconditions here too, similar to the previous scenario.
Suggested:
Goal: Force mgmt-agent’s SwiftNICController to re-query the Azure Compute API and republish the node’s SWIFT NIC capacity, without disrupting workloads already running on the node.
Preconditions: Confirmed via node status that node.status.capacity[‘aro.openshift.io/swift-nic’] is 0 or missing, while the VMSS instance itself shows secondary NICs attached in Azure. Worth noting explicitly that this restart uses Server-Side Apply per the controller code, so it should only resync existing state rather than disrupt running pods -stating that up front would give the on-call SRE more confidence before running it.
| kubectl rollout restart deployment -n mgmt-agent | ||
| ``` | ||
|
|
||
| ## Scenario: CNS not running on a node (HCP side) |
There was a problem hiding this comment.
Same as the other scenarios above — missing Goal and Preconditions before the fix steps.
Suggested:
Goal: Restore the CNS pod to Running so it resumes handing out IPs on the node.
Preconditions: CNS pod confirmed not Running (CrashLoopBackOff, Pending, or missing entirely).
|
|
||
| Escalate to the Azure AKS team if CNS cannot be restarted or the crash is not caused by HCP configuration. | ||
|
|
||
| ## Scenario: Azure platform issue (NRP / DNC / AKS RP regression) |
There was a problem hiding this comment.
Same as the other scenarios above — missing Goal and Preconditions before the escalation steps.
Goal: Escalate to the correct Microsoft team once ARO-HCP-side causes have been ruled out.
Preconditions: Prior diagnostic steps found no ARO-HCP-owned root cause, and Azure platform signals (AKS/NRP) are showing an anomaly.
|
|
||
| > **TODO:** Document the exact process for Red Hat SREs to open an IcM against Microsoft teams (AME account requirements, IcM queue IDs). Confirm with #external-wg-aro-hcp. | ||
|
|
||
| # Validation and Confirmation |
There was a problem hiding this comment.
Nice checklist here. One addition worth considering: this confirms internal system state (pods, metrics, alerts, node status), but doesn't explicitly loop back to confirm the customer-visible symptoms from earlier in the TSG have cleared — specifically, webhook failures and cert rotation issues were both called out as customer-visible symptoms. Might be worth a line confirming those two specifically resolved, since they're what originally indicated customer impact.
| # After Incident | ||
|
|
||
| ## Postmortem Notes | ||
|
|
There was a problem hiding this comment.
Good postmortem checklist here. One gap: nothing says when this actually needs a formal PMR versus just these notes. Could we add a line on that, plus a link to the PMR process if we have one?
…O-25382) Add User Journey, TSG, and alerting for SWIFT networking on ARO-HCP management clusters. User Journey (new 4-section template format): - Journey: customer and SRE goals for private HCP connectivity - Architecture: mermaid diagram of happy-path NIC assignment flow, component table, glossary of SWIFT-specific terms - SLIs/SLOs/TSG Routing: single table mapping each SLI to its SLO, alert names, and TSG entry point - Escalation: per-team blocks for Cloudnet/NRP, Azure AKS, NMAgent TSG (8-section template): - Alert triage table routing each alert to a diagnostic step - Cluster access (JIT + hcpctl + breakglass guide) - Glossary, full severity table (Platform Incident + CRI paths) - Diagnostic steps: blast radius, HCP-side prereqs, CNS log classification, AKS Kusto queries, Azure platform signals - Mitigation scenarios: stuck MTPNC, mgmt-agent NIC capacity, CNS - Escalation with evidence-to-attach per team - CNS Prometheus metrics and recording rules appendices Alerting and recording rules: - Recording rules for router pod startup latency p99 (kube-state-metrics) and CNS IP assignment latency p99 (CNS PodMonitor) - Alerts: userJourneySwiftLatencyP99 (fast/medium/slow burn), userJourneySwiftErrors (fast/medium burn), userJourneySwiftCNSLatencyP99 (fast/medium burn), SwiftCNSAvailability3d, SwiftPendingProgramming - promtool test coverage for all alerts - CNS PodMonitor for scraping azure-cns metrics Design docs (drafts, not published): - uj-alerts-swift.md: alert naming, routing lanes, burn-rate tiers - uj-slislo-swift.md: SLI design rationale, recording rule chain - uj-dashboard-swift.md: Grafana dashboard panel specifications
98bedfc to
30f1ac0
Compare
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: ventifus The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
351ef4d to
5025564
Compare
- Fully spec konnectivity tunnel health SLIs (stream error rate, dial failure rate) across all design docs: SLI/SLO, alerts, dashboard, and implementation spec - Fix TSG cross-references: "Step 4" -> "Step 5" matching anchors - Rewrite implementation spec: correct file names, alert name prefix, add konnectivity recording rules and alerts - Add slow-burn tier (userJourneySwiftLatencyP993d) to alerts doc - Add konnectivity panels to dashboard design doc - Add konnectivity-server PodMonitor (port 8093)
5025564 to
35521d8
Compare
3 participants
https://redhat.atlassian.net/browse/ARO-25382
What
User journey, TSG, and alerting for SWIFT networking on ARO-HCP management clusters.
rp-rules.bicepuserJourneySwiftLatencyP99{1h5m,6h30m}userJourneySwiftErrors{1h5m,6h30m}userJourneySwiftCNSLatencyP99{1h5m,6h30m}userJourneySwiftKonnectivityErrors{1h5m,6h30m}userJourneySwiftKonnectivityDialFailures{1h5m,6h30m}SwiftCNSAvailability3dSwiftPendingProgrammingWhy
SWIFT provides the private network path between customer VNets and hosted control planes. When it fails, worker nodes go NotReady, webhooks break, and cert rotations fail. This PR adds observability at two layers:
Test plan
make alertsandmake recording-rulespass; all promtool tests green